Integrated virtual simulation and face-to-face simulation for clinical judgment training among undergraduate nursing students: a mixed-methods study

Background Virtual simulation and face-to-face simulation are effective for clinical judgment training. Rare studies have tried to improve clinical judgment ability by applying virtual simulation and face-to-face simulation together. This study aimed to evaluate the effect of an integrated non-immersive virtual simulation and high-fidelity face-to-face simulation program on enhancing nursing students’ clinical judgment ability and understanding of nursing students’ experiences of the combined simulation. Methods A sequential exploratory mixed-methods study was conducted in a nursing simulation center of a university in Central China. Third-year nursing students (n = 122) taking clinical training in ICUs were subsequentially assigned to the integrated non-immersive virtual simulation and high-fidelity face-to-face simulation program arm (n = 61) or the face-to-face simulation-only arm (n = 61) according to the order in which they entered in ICU training. Clinical judgment ability was measured by the Lasater Clinical Judgment Rubric (LCJR). Focus group interviews were conducted to gather qualitative data. Results Students in both arms demonstrated significant improvement in clinical judgment ability scores after simulation, and students in the integrated arm reported more improvement than students in the face-to-face simulation-only arm. The qualitative quotes provided a context for the quantitative improvement measured by the LJCR in the integrated arm. Most of the quantitative findings were confirmed by qualitative findings, including the domains and items in the LJCR. The findings verified and favored the effect of the combination of non-immersive virtual simulation and high-fidelity face-to-face simulation integrated program on enhancing nursing students’ clinical judgment ability. Conclusions The integrated virtual simulation and face-to-face simulation program was feasible and enhanced nursing students’ self-reported clinical judgment ability. This integrated non-immersive virtual simulation and high-fidelity face-to-face simulation program may benefit nursing students and newly graduated nurses in the ICU more than face-to-face simulation only. Supplementary Information The online version contains supplementary material available at 10.1186/s12909-023-04988-6.


Background
Clinical judgment ability refers to applying knowledge and professional skills to a clinical situation to solve problems and is a core competency that nursing students should possess before transited to registered nurses [1,2].With the high acuity of patients' conditions and various public health emergencies, the complicated care environment requires nurses to employ good judgment ability to maintain patients' safety and improve patients' outcomes, which requires rapid assessment and judgment of patients so that patients receive timely and effective care [3,4].Nursing students take advantage of clinical training in various clinical settings, including intensive care units (ICUs), to enhance their clinical judgment ability [5].Massive lectures and limited clinical placement opportunities impede nursing students' immersion in adequate clinical experiences [6], thus hindering the training of clinical judgment ability among nursing students.
With advances in technology, new approaches to nursing education are being used in nursing curricula, such as simulation and virtual simulation, to improve the clinical judgment skills of nursing students [7,8].Several benefits of high-fidelity simulation have already been established, including enhanced performance during simulated resuscitation and improved triage skills [9,10], enhanced nursing students' communication, teamwork, and ability to manage complex situations [11,12].A virtual simulation is an innovative approach to providing virtual clinical experiences through a digital platform where learners can complete specific tasks in various potential environments, use the information to provide assessment and care, make clinical decisions, and observe the outcomes of actions [13,14].Virtual simulation allows nursing students to familiarize themselves first with problems encountered in simulation by providing flexible and repetitive exercises [15].Although previous studies have the effect of simulation-based intensive care nursing training on enhancing nursing students' performance [9,12], and the impact of virtual simulation training on improving clinical judgment ability in newly graduated nurses [16], rare studies have tried to improve clinical judgment ability by integrating virtual simulation and faceto-face simulation together among nursing students during the clinical training in ICUs.
The present study aimed to verify the effect of a nonimmersive virtual simulation and high-fidelity face-toface simulation integrated program and compared the effect between the intergrade program (Integrated) and the face-to-face stimulation only (Simulation) on enhancing the clinical judgment ability of third-year nursing students during clinical training in ICUs.One of the vSim for Nursing scenarios (Version 1, Wolters Kluwer, Philadelphia, PA, USA) was used in the virtual simulation (acute pulmonary embolism); this scenario (acute pulmonary embolism) was also adopted into high-fidelity face-to-face simulation scenario in the current study following the standards of best practice for simulation [17].The hypotheses included: a) both Integrated program and Simulation could improve students' clinical judgment ability, b) students in the Integrated arm improve more than students in the Simulation arm, and 3) the simulation was acceptable among nursing students.This study also explored students' experiences of the integrated program to provide context for the effects of virtual simulation.

Study design
An explanatory sequential mixed methods design (QUAN → qual) [18] was employed in the present study.This design consisted of quantitative analysis and qualitative description to enrich and provide context for the quantitative findings.The guideline for reported mixed-methods studies [19] was followed (Supplementary file 1).A two-group, non-randomized, pre-and post-test quasi-experiment design was conducted in the quantitative strand (Fig. 1).Focus group interviews were conducted sequentially in the qualitative strand.Inform consent was signed by each of the participants following ethics approval.

Quantitative phase Setting
This study was conducted in a simulation center in central China.The Bachelor of Science in Nursing program is a four-year synchronous lecture and clinical training program.Simulation has been embedded in most nursing curricula, including health assessment, fundamentals of nursing, and clinical nursing (i.e., medical-surgical nursing, pediatric nursing, intensive care nursing).Undergraduate nursing students take some lectures and take turns completing assigned clinical training in each academic semester.The four-week clinical training in ICU was appointed in the third year, and students take turns to complete the ICU training in Fall and Spring semesters.

The clinical training in ICU
The clinical training in ICU in the third year consisted of lectures, simulations, and clinical placement in each of the four weeks.During each week, students took a pre-test on Day 1, and a lecture was then delivered by a certificated simulation lecturer (JC).Students took turns to run face-to-face simulation scenarios on Day 2 led by a certificated simulation lecturer (JC), the students also spent three hours in the skill lab to practice related nursing skills, including but not limited to blood draw, venipuncture, and oral suction on Day 2. Students were then placed in ICU on Day 3 with an assigned RN and patients; a debriefing was conducted at the end of the day.Students were taking other courses on Day 4 and 5.

Participants
Two cohorts of third-year nursing students (Cohort 2013 and Cohort 2014) taking their four-week training in the ICU from September 2015 through June 2017 were invited to join this study.After acquiring informed consent, one hundred twenty-two students Fig. 1 Flow chart of the study.Note: Integrated, the non-immersive virtual simulation and high-fidelity face-to-face simulation integrated program arm were divided into sixteen groups, with eight groups in each cohort (five to eight students in each group, supplementary Table 1).Students were subsequently assigned to the Integrated and Simulation arms (supplementary Table 1) according to the order in which they entered the ICU internship.Eight groups of students (n = 61) were assigned to the Integrated arm, and the other eight groups of students were in the Simulation arm (n = 61).

The vSim
Students in the Integrated arm were provided a user ID and password to access the vSim for Nursing scenario after the pre-test.The scenario, "acute pulmonary embolism, " from vSim for Nursing (Version 1, Wolters Kluwer, Philadelphia, PA, USA) was chosen for the students to complete during the third week of the ICU training.The vSim for Nursing scenario and simulation scenario focus on the initial identification of the signs and symptoms of acute pulmonary embolism post-surgery.The nursing process, including timely assessment, communication with physicians, drug administration, post-assessment, and health education, was also the critical point of this scenario.Students were encouraged to try as many times as they liked and were asked to provide a screenshot to show that they achieved a score of at least 90 out of 100 before the face-to-face simulation on Day 2. Students were told to refrain from discussing the vSim for Nursing nor simulation with each other out of the ICU internship.

The face-to-face simulation
Certificated simulation lecturers adopted one scenario (acute pulmonary embolism) from vSim for Nursing (Version 1, Wolters Kluwer, Philadelphia, PA, USA) into a simulation scenario following the standards of best practice for simulation [17].The lecture team developed the other three face-to-face simulation scenarios following the same standards of best practice, including cases with septic shock, lung cancer, and trauma.Each of the four scenarios was delivered in the four weeks of the ICU training.The simulation scenario was repeated three times, each lasting for one hour, including 15 min of introduction and pre-discussion, 15 min of running the scenario, and 30 min of debriefing after the simulation following our previous protocols [9,10].Two or Fig. 2 Data collection procedure.Note: Integrated, the non-immersive virtual simulation and high-fidelity face-to-face simulation integrated program arm three students (5 to 8 students in each group, supplemental Table 1) took turns to immerse in the simulation scenario.

Clinical judgment ability
The primary outcome variable, students' clinical judgment ability, was assessed by the 11-item Lasater Clinical Judgment Rubric (LCJR) [16].The 11-item LCJR includes effective discovery processes (3 items), valid interpretation of the material (2 items), effective feedback was given (4 items), and effective reflection (2 items).The total score of the LCJR ranges from 11 to 44, with each item rating on a 4-point Likert scale; a higher LCJR score indicates higher self-reported clinical judgment ability.The Cronbach's alpha of LCJR was 0.953 in the current study.

Simulation Design Scale (SDS)
There are 20 items and five domains in this Likert fivepoint scale, with 5 items in objectives/information, 4 in student support, 5 in problem-solving, 2 in guided reflection and feedback, and 2 in fidelity [20].The total mean score of SDS ranges from 1 to 5, with each item ranging from 1 to 5 (1 = strongly disagree, 5 = strongly agree).All mean scores greater than 3.5 were considered relatively high for simulation design.The Cronbach's alpha of the SCS was 0.902 in the current study.

Educational practices in simulation scale
There are 16 items and four domains in this Likert fivepoint scale, with 10 items in active learning, 2 in collaboration, 2 in diverse ways of learning, and 2 in high expectations [21].The total mean score of this scale ranges from 1 to 5, with each item ranging from 1 to 5 (1 = strongly disagree, 5 = strongly agree).A mean score greater than 4 indicates a high perspective of education practice in simulation.The Cronbach's alpha was 0.902 in the current study.

Student satisfaction and self-confidence in learning
There are 13 items and two domains in this Likert fivepoint scale, with 5 items in satisfaction and 8 in self-confidence in learning [21].The total mean score of this scale ranges from 1 to 5, with each item ranging from 1 to 5 (1 = strongly disagree, 5 = strongly agree).A higher mean score indicates a higher level of satisfaction and self-confidence in learning.The Cronbach's alpha was 0.889 in this study.

Sample size calculation
The enhanced clinical judgment ability measured by the LCJR was the primary outcome.The sample size for each arm was calculated for a power and a false discovery rate set at 0.8 and 0.05, respectively.The sample size calculations were performed using G*Power 3.1 [22,23].For the first hypothesis, we assumed that Simulation has a median effect (Cohen's d = 0.5) [24] on improving clinical judgment ability among nursing students, thus a significant improvement in scores of self-reported clinical judgment ability from pre-test to post-test; thirty-four participants in the Stimulation arm were estimated to test the difference in this assumption.We also hypothesized that the Integrated arm improved more than students in the Simulation arm; fifty-one participants in each arm were estimated to test the difference in this assumption with an effect size of 0.50 (Cohen's d) [24].In summary, at least 102 subjects should be recruited (51 in each arm) to achieve a power of 0.8 at a significance level of 0.05.

Data analysis
IBM SPSS Statistics version 25 (IBM Inc., Armonk, NY, USA) was used for data analysis.The frequency, percentage, mean, and standard difference were used for the statistical description of pre-and post-test scores.Pre-and post-test LCJR scores in each arm were compared using a paired t-test to verify the effect of the Integrated program and Simulation on improving students' clinical judgment ability.The difference between the pre-test LCJR scores, and the post-test LCJR scores from the Integrated and the Simulation arms was compared using an independent t-test to explore if students in the Integrated arm improved more on clinical judgment ability than students in the Simulation arm.The difference between the evaluation for the simulation study from the Integrated and the Simulation arms was compared using an independent t-test to investigate if the students in the two arms perceived differently of the face-to-face stimulation.

Qualitative phase
The qualitative phase used focus group interviews to collect students' data.The focus group aimed to a) explore novel findings that emerge in the analysis of the quantitative data, b) identify new areas of inquiry in addition to the quantitative assessments, and c) assess the validity of quantitative findings by presenting them for interpretation to study participants in qualitative interviews.Students in the Integrated arm were invited to join the focus group in the qualitative phase after the post-test in a private room.The interview protocol was grounded in the quantitative results from the first phase of the study (Supplementary file 2).To maintain a heightened level of awareness, a journal of personal feeling reflection, and contemplation was kept throughout the study.Eight focus group interviews were conducted, and each interview was audiotaped and transcribed verbatim by a research assistant.Semi-structured interview data analysis followed the process: (a) preparing the data for analysis, (b) conducting content analysis [18], and (c) transforming the textual data into numerical data if possible [25].This process facilitated the integration of quantitative and qualitative data in a mixed-methods design.

Integrating the quantitative and qualitative findings
The results of the quantitative and qualitative phases were integrated while comparing the results of the entire study, as well as in the dissection of the study (Fig. 1 for a diagram of the explanatory sequential mixed methods design).Themes extracted from the qualitative phase were compared and contrasted with the subjective performance and clinical judgment measurements in the quantitative phase [18].Similarity and differences across and with quantitative and qualitative data were presented in tables.

Demographic information of the included students
Among the 122 students invited, 105 were females, and 17 were males, aged 19 to 24 (20.65 ± 0.832) years old.All the students invited agreed to participate, on one dropped out.There were no significant differences between the two groups regarding gender and age (p = 0.562).

Quantitative strand findings The effect of integrated program and simulation on clinical judgment among nursing students
The scores of clinical judgment ability measured by the LCJR before and after the simulation were presented in Table 1.There was no significant difference regarding the pre-test scores acquired from students in the Integrated and Simulation arms.Both the Integrated and Simulation arms showed significant improvement after the simulation (all p < 0.01).Regarding the difference between the Integrated arm and the Simulation arm at post-test, scores in the Integrated arm were significantly higher in five items of two domains (all p < 0.05), including effective discovery process and valid interpretation of the material.No significant difference was observed in the other six items of the other two domains (effective feedback was given and effective reflection).
The mean difference of scores between pre and posttest in each item and domain of the LCJR in the two arms were also compared (Table 2).Scores in the Integrated arm significantly improved more than those in the Simulation arm in seven of the 11 items, and three of the four domains of the LCJR, including effective discovery processes (t = 4.82, p < 0.05), valid interpretation of the material (t = 5.01, p < 0.05), effective reflection (t = 2.10, p < 0.05).

Nursing students' feedback on simulation design
Table 3 presents students' feedback on the simulation in the two arms.All domains in the SDS were rated relatively high both in the Integrated and the Simulation arms, and the difference between these two arms was not significant.However, compared to the Simulation arm, the Integrated arm has higher mean scores in the domains of fidelity (4.09 (SD = 0.588) vs. 3.81 (SD = 0.881), p = 0.042).
Mean scores of educational practice in simulation and each domain were greater than 4 in both arms.Collaboration was the highest, followed by diverse ways of learning, high expectations, and active learning.There was no significant difference between these two arms regarding mean scores of educational practice in simulation and each domain.
Mean scores in satisfaction and self-confidence of students were also greater than 4, with higher scores in  the satisfaction domain than in self-confidence in both arms.Regarding scores in satisfaction and self-confidence of students in the two arms, the difference was not significant.

Qualitative strand findings
The findings of focus group interviews supported the improvement of clinical judgment and helped interpret quantitative data related to these improvements.The vSim for Nursing allowed students to reflect on their decisions in a safe and self-controlled learning environment.Themes and quotes consistent with the domains of the LJCR are listed below and in Table 4.Other advantages, inadequacies, and recommendations were also summarized (Table 4).

Effect on improving clinical judgment Effective discovery processes
Through step-by-step feedback and timely reminders in vSim practice, students can identify the critical learning points and important questions they must know.

"I think the tone and scene in vSim are authentic; it (vSim) can lead you to believe what to pay attention to when dealing with patients and their diseases. You do not know what to observe initially, but later you know what to discover because the feedback from vSim will lead you to step by step.
"vSim allows us to pay attention to the information not covered in the simulation, such as calculating the dosage of drugs and the time interval between drugs.We usually don't pay much attention to the doctor's orders when we do the simulation, so I think combining vSim and simulation is better than just doing a simulation".

Valid interpretation of the material
Students believed they could explain and sort out the materials most relevant to patients and evaluate and rank the information when dealing with complex scenarios.
"vSim is a bit strict, but you can know the rules as you need to inquire about allergies before using heparin and the rules of checking, and then we'll apply this to the actual (drug administration) process.However, simulation can also be more flexible.For example, in vSim, when we evaluate pain, we will be with the patients in the private room when we ask questions here and there (stay with the patient).Still, in the actual process, it will be very chaotic (chaotic environment in clinical wards), so the two can be combined with and complementary".

Effective feedback was given
Students could adjust the treatment regimen on timebased on the information collected by themselves and the feedback from patients and compare them with the knowledge and experience they acquired from previous learning to develop an intervention program.
"vSim can give effective feedback quickly.Let you know how to improve; there are many opportunities; there is no real tension; quickly master its process." "…..do vSim you feel like a combination of examination and game, there is a sense of painting, both image memory, but also deepen the body memory".

Effective reflection
Students could benefit from the timely reflection provided by the vSim.
"There is no such thing as practicing on real people; we can repeat many times.Suppose there is a violation of nursing principles or some ideas.In that case, there will be reminders on it, as well as sources, that is, we can systematically understand the relevant knowledge and target points of a disease".

Other advantages The subjective initiative of nursing students was improved
The vSim enhanced students' learning interests.
"Read the book and then do vSim, and you will sometimes find that the knowledge is incomplete.When I get stuck in my mind, I will be more impressed if I turn to a book".

The teaching form of the course is novel and flexible
The vSim allowed students to choose the time and place freely.The combination of vSim and simulation led students to think actively, stimulated their interest in learning, and made the boring skill training more vivid.

"I think vSim can save time because we can practice vSim in the dormitory; we do not need to gather in the classroom; sometimes it's hard to find an available time slot for everyone in the group. "
"vSim provides us with such a platform.Without vSim, our main experience might come from clinical and classroom teaching.Now with vSim, we can practice alone without the teacher or in the clinic, which is a relatively easy experience".

Advantages
The vSim for Nursing improved students' learning interest, leaded students to think actively, and made the boring skill training more vivid

Inadequacies
There was no oral communication and in-person interaction during the vSim for Nursing practice In the vSim for Nursing process, there is no communication with patients, there is no real interaction

Recommendation
Students suggested improving the design and the fidelity and creating more scenarios The design of vSim for Nursing itself needs to be closer to the clinical setting Only one scenario may make you feel boring.I think more virtual simulation scenarios should be developed

Inadequacies and recommendation
Students mentioned that there was no real communication and interaction during vSim practice.
"In the virtual simulation process, there is no communication with patients; there is no real interaction, can not respond naturally like people.Most of the time, as a reminder, the answer is always a state of emergency." Themes of recommendations to improve the design to combine vSim and simulation also emerged, such as improving the fidelity and creating more scenarios.

"The design of vSim itself needs to be closer to the clinical setting, to increase the auscultation, palpation and abnormal signs measurement function through computer technology. "
"Only one scenario may make you feel bored.I think more vSim scenarios should be developed."

Mixed methods findings
The mixed-methods findings integrated quantitative and qualitative feedback of clinical judgment among nursing students (Table 5).The qualitative quotes provided a context for the quantitative improvement measured by the LJCR.Most of the quantitative findings were confirmed by qualitative findings, including the domains and items in the LJCR.The qualitative quotes expanded quantitative improvement in the domain of "valid interpretation of the material".

Discussion
The feasibility and effect of a virtual simulation and simulation for improving nursing students' self-reported clinical judgment ability were established in this mixedmethods study.The findings in the quantitative strand supported all three hypotheses by comparing and verifying the effect of a combination of virtual simulation and face-to-face simulation.The focus group interviews in the qualitative strand yielded different but complementary data, confirming findings in the quantitative strand and providing a context for the quantitative findings.
Improvement after simulation in both arms supported the first hypothesis that both the Integrated program and simulation could enhance nursing students' self-reported clinical judgment ability (all p < 0.01, Table 1), which is in line with previous studies [10,[26][27][28].Clinical judgment is the process by which the nurse gathers information, identifies vital information, identifies problems, sets improvement goals, plans and implements interventions, evaluates results, and learns to reflect on them.Participation in simulation requires nursing students to be involved in the process and employ their skills to solve the problems, confirmed by previous studies [9,10].
Students in the Integrated arm improved more than students in the Simulation arm in scores of effective discovery processes, valid interpretation of the material, and effective reflection (all p < 0.05, Table 2), favored the Integrated program and verified our second hypothesis.Table 2 shows the average differences in LCJR scores between pre-and post-test among the two arms, demonstrating that students in the combination of vSim for Nursing and simulation arm reported significantly more improvement in clinical judgment ability than those in the simulation-only arm.These results were consistent with the interview results (see Table 5).Students mentioned in the interview that the combination of vSim for Nursing and simulation could improve communication and operation skills, achieve effective intervention, and improve self-confidence.Evidence showed that students' confidence levels improved with repeated simulation [29].In addition, previous study shows that students tend to show tension, anxiety, and fear when being assessed and tested in a simulated environment due to the expectation of achieving near-perfect execution [8].Repeated training in virtual simulation may relieve students' stress before they immerse in the simulation.However, we did not record the times each student tried the vSim scenario as well as the final score of the vSim scenario, since students in the Integrated arm were provided ample opportunity to get familiar with the vSim scenario, an individual style of learning was also the benefit of the vSim.
Our qualitative findings provided context for the improvement of clinical judgment ability in the Integrated arm.The qualitative data supported the significant improvement in effective discovery processes in the Integrated arm found in the quantitative strand, consisting of focused observations, recognizing deviations, and information seeking (Table 5).Students in the interview session also mentioned that repeatedly conducting vSim for Nursing was like experiencing multiple simulations that enabled them to continuously summarize experiences and rules, supported by a previous study [30].The vSim for Nursing plays a suggestive role in subsequent simulation, allowing students to collect targeted information, discover outliers, and accurately evaluate the implementation of practical measures, ultimately gradually translating knowledge into clinical thinking and clinical judgment [31].
Mixed-methods findings also supported our hypotheses.The significant improvement of "valid interpretation of the material" and "effective reflection, " especially the prioritizing data, was also mentioned in the interview (Table 5).The sequential evaluation and operation  of vSim for Nursing include standardized diagnosis and treatment procedures, and the severity of each error is graded and marked, which enables students to think, distinguish between priorities in the processing process, pay attention to the principle of the problem, and to respond orderly and calmly.In the vSim for Nursing scenario, students interpreted the collected information for further treatment.Thus, virtual simulation helped students think, understand, and apply their previous knowledge [32].The vSim for Nursing animation is vivid so that students feel different from the classroom teaching [33].
Through the immersive learning of vSim for Nursing, students deepen their multidimensional understanding [34].Identical to previous findings, this study allowed students to perform virtual exercises repeatedly ondemand before simulation until they were satisfied with their performance [35].Even though students in the two arms rated different scores in support and fidelity in simulation design, they all rated relatively high when considering their perspective of the simulation, which supported the acceptability of the simulation scenario adopted from the vSim for Nursing, and also supported our third hypothesis.Virtual simulation could be an alternative to improve the fidelity since virtual-reality simulation has the advantage of an entire virtual 3D Platform to present live-action training as closely as possible [31,36].Students admitted that the learning objectives were met, they were satisfied with the simulation, and they rated pretty high for simulation design, education practice in simulation, and self-confidence in learning.The vSim for Nursing coupling simulation education for ICU internship bolstered nursing students' clinical judgment ability by facilitating the translation of knowledge to practice.Maximizing the opportunity for students to participate in clinical judgment that cannot be engaged in clinical practice and to experience clinical case management by engaging in virtual simulation may be an addition to classroom lectures to improve learning outcomes [37].
However, the post-simulation improvement of "effective feedback was given" between the Integrated and the Simulation was not significantly different (Table 2).The main reasons could be that the vSim for Nursing lacks a flexible communication environment, a design to develop students' hands-on ability and real-time observable changes in patients' conditions.In addition, the vSim was independently practiced by students without a faculty member; the feedback was generated by the system and may have lacked depth.The other reason could be the short training time since the simulation only lasted 1 h.

Implications
The findings in the current study added a new dimension to nursing education's transition from the face-to-face paradigm to a learner-centered virtual simulation-based education.Although conducted before COVID-19 when in-person learning and face-to-face was the predominant teaching paradigm, this study was particularly important in the context of limited in-person contact during the COVID-19 pandemic and post-pandemic [38].This study also provided suggestions on how virtual simulation strategies might be improved and what might be developed in future studies through focus interviews.Further studies could also create more virtual reality scenarios since only one scenario was employed in the current study.Only non-immersive virtual simulation was applied in the current; further studies may also develop semi-immersive or immersive virtual simulation and investigate the effect of this virtual simulation on healthcare education.This study only included nursing students taking clinical training in ICU; studies may also explore the effect of the integrated virtual simulation and face-toface simulation program among nursing students receiving training in other clinical settings.

Limitations
Several limitations of this study need attention.First, the study period ranged from 2015 to 2017 though we controlled other variables, i.e., using the same teaching material, same instructor, and same settings of the simulation.We used convenience sampling instead of a randomized controlled design, and all students participated in the simulation, which may cover the effect of virtual simulation (vSim for Nursing).In addition, this study was conducted in one center.The clinical judgment ability was self-reported other than acquired by objective assessment, which may impact the findings, and the improvement does not guarantee improvement of patients' outcomes.The improved clinical judgment ability was measured immediately after the simulation; the improvement may fade after long periods.We did not compare the two arms in terms of the scores on acute pulmonary embolism and the amount of time spent in the faceto-face simulation.A comparison of the score and the amount of time spent in the simulation may provide more information regarding the effect of the integrated virtual simulation and face-to-face simulation program.Further multicenter randomized controlled studies may recruit a larger sample to validate the current findings.Other studies may also consider adding an objective assessment of clinical judgment ability as well as including patients' outcomes if possible.

Conclusions
This mixed-methods study comparatively verified the effect of a combination of virtual simulation and faceto-face simulation for enhancing nursing students' selfreported clinical judgment ability.Quantitative data analysis supported the clinical judgment improvement, and qualitative data confirmed the quantitative findings.The integrated virtual simulation and face-to-face simulation program provides a forum to practice clinical judgment skills such as critical thinking, communication, and decision-making.Findings from this study may also benefit new graduate nurses in ICU since the scenario was developed for nursing internships in ICU.

Figure 2
Figure 2 illustrates the detailed procedure on the timeline in the quantitative phase.All students completed a pre-knowledge test of acute pulmonary embolism and self-assessment of clinical judgment ability on Day 1 in the third week of the ICU training before the lecture.Only students in the Integrated arm were additionally invited to interact with the vSim for Nursing scenario as a flipped-learning strategy after the pretest on Day 1.The face-to-face simulation section as a standard component of the ICU training was provided to students in both the Integrated and Simulation arms on Day 2. After the face-to-face simulation on Day 2, a post-knowledge test of acute pulmonary embolism, self-assessment of clinical judgment ability, and evaluation of the simulation were conducted among all students.

Table 1
Comparison of nursing students' scores in each item and domain of the Lasater Clinical Judgment Rubric (LCJR) between preand post-test in two arms (n = 122) Integrated, the non-immersive virtual simulation and high-fidelity face-to-face simulation integrated program arm a comparison of scores in post-test between the Integrated arm and Simulation arm by independent samples t-test ** compared with scores in pre-test by paired samples t-test, p < .001

Table 2
Comparison of the mean difference of participants' scores in each item and domain of the Lasater Clinical Judgment Rubric (LCJR) between pre and post-test in two arms (n = 122) Integrated, the non-immersive virtual simulation and high-fidelity face-to-face simulation integrated program arm

Table 3
Students' perspective of the simulation (n = 122) Integrated, the non-immersive virtual simulation and high-fidelity face-to-face simulation integrated program arm

Table 4
Qualitative findings from focus group interviews in the Integrated arm (n

Table 5
Integration of clinical judgment improvement from quantitative and qualitative data among nursing students in the Integrated arm (n

Table 5 (continued) Quantitative findings (mean score at pre-and post-test) Qualitative findings (quote) Mixed methods interpretation
Integrated, the non-immersive virtual simulation and high-fidelity face-to-face simulation integrated program arm